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Abstract 



In this paper we are concerned with the number of nonnegative solu- 
tions of the elliptic system 



(P) 



— Ail = Q u (u, v) + -^rH u (u, v), in Q, 
—Av = Q v (u, v) + ^rH v (u, v), in Q, 
u — v — 0, on dtt, 



where SI C R is a bounded smooth domain, N > 3, 2* := 2N/(N - 2) 
and Q U ,H U and Q v , H v are the partial derivatives of the homogeneous 
functions Q, H € C^R^R), where R+ := [0, oo) x [0, oo). In the proofs 
we apply variational methods and Ljusternik-Schnirelmann theory. 
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1 Introduction 



In this paper we are concerned with the number of nonnegative solutions of the 
elliptic system 



(P) 



-Au = Q u (u,v) + ±-H u (u,v), in Q, 
-Av = Q v (u, v) + ^rH v (u, v), in 0, 
u = v = 0, on dfl, 



where fl C R N is a bounded smooth domain, N > 3, 2* := 2N/(N - 2) and 
Q U ,H U and Q v , H v are the partial derivatives of the homogeneous functions 
Q, H e C^K^K), where M 2 . := [0, oo) x [0,oo). 

We are interested in the case that H has critical growth. More specifically, 
the assumptions on H = H(s,t) are the following. 

(Hq) H is 2*-homogeneous, that is, 



H(0s, Ot) = 2 H(s, t) for each 9 > 0, (s, t) e 



(ffi) H a (0,l) = 0,fl t (l,0) = 0; 

(iJ 2 ) H(s,t) > for each s,t > 0; 

(i? 3 ) ff a (s,t) > 0, i? t (s,i) > for each (s,t) e 

(i?4) the 1-homogeneous function (s,t) i-> H(s 1 / 2 * , t 1 / 2 * ) is concave in ; 

The function Q = Q(s, t) is a lower order perturbation term satisfying 
(Qo) Q is ^-homogeneous for some 2 < q < 2*; 
(Qi) Q 8 (0,1) = 0, Q t (l,0) = 0. 

In order to present our results we introduce the following numbers 

\i := min {Q(s, t) : s q + t q = 1, s, t > 0} (1.1) 

and 

A := max {Q(s, t) : s q + t q = 1, s, t > 0} . (1.2) 

We say that a weak solution z — (u, v) e ^o(^) x (^) °^ P r °blem (P) is 
nonnegative if u, v > in ft. If Y is a closed set of a topological space Z, we 
denote by catz(Y") the Ljusternik-Schnirelmann category of Y in Z, namely the 
least number of closed and contractible sets in Z which cover Y. We are now 
ready to state our first result. 

Theorem 1.1 Suppose that H satisfies (Hq) — (H4,) andQ satisfies (Qo)-(Qi)- 
Then there exists A > such that the problem (P) has at least catn(f2) nonzero 
nonnegative solutions provided A, (i G (0, A). 



2 



In the proof we apply variational methods, Ljusternik-Schnirelmann theory 
and a technique introduced by Benci and Cerami [3j- It consists in making 
precise comparisons between the category of some sublevel sets of the associated 
functional with the category of the set ft. In order to overcame the lack of 
compactness due to the critical growth of H we use the ideas of Brezis and 
Nirenberg [3], besides the paper of Morais Filho and Souto [12] , where it is 
proved that the number 




(1-3) 

plays an important role when dealing with critical systems like (P) . Actually, we 
use the above constant and adapt some calculations performed in Myiagaki [14] 
to localize the energy levels where the Palais-Smale condition fails. We would 
like to mention that, as a byproduct of our arguments, we extend the existence 
result of [12] for any subcritical degree of homogeneity of the perturbation Q 
(see Theorems l2~4l and [23]) . 

Notice that condition (Qi) discard examples like Q(s,t) = s g + t q + st 9 ^ 1 
since, in this case, <5s(0,l) = 1. However, we can also consider this situation 
if the subcritical perturbation satisfies q > 2. More specifically, the following 
holds 

Theorem 1.2 Suppose that H satisfies (Hq) — (H4), Q satisfies {Qo) with q > 2 
and 

(Qi) Q S (0,1) > andQ t {l,0) > 0. 

Then there exists A > such that the problem (P) has at least catn(f2) nonzero 
nonnegative solutions provided A, /i S (0, A). 

The difference when dealing with (Qi) or (Qi) is just in the way we extend 
the function Q to the whole R 2 . Since we want to apply minimax methods this 
extension needs to be made in a smooth way. We refer to the beginning of the 
next section for more details about the possible extensions. 

Concerning the class of nonlinearities we are considering, we present in Sec- 
tion [5] some examples of functions satisfying our hypothesis. There, we also 
make some comments about the possibility of proving that the solutions are 
positive in O and we state other settings in which our results hold, including 
the possibility of having a sum of subcritical terms with different degrees of 
homogeneity. As a final remark, we would like to mention that the theorems 
remain valid for N — 3 if the degree of homogeneity of Q satisfies 4 < q < 6 
(see Remark [2~3|) . 

The starting point on the study of the system (P) is its scalar version 

- Au = 0\u\ q - 2 u+ \u\ r ' 2 u inQ, ueJ?o(fl), (1.4) 

with 2 < q < 2*. In a pioneer work Brezis and Nirenberg [4] showed that, for 
q = 2, the existence of positive solutions is related with the interaction between 
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the parameter 9 with the first eigenvalue 0i(f2) of the operator (— A, Hq (0)). 
Among other results they showed that, if g = 2, the problem has at least one 
positive solution provided N > 4 and < 9 < They also obtained some 

results for the case 2 < q < 2*. 

After the paper of Brezis and Nirenberg, a lot of works dealing with critical 
nonlinearities have been appeared. Concerning the question of multiplicity, we 
recall that Rey [15] and Lazzo [5] proved that, for q = 2, the problem (| 1 .4[) has 
at least cato(Q) positive solutions (see also the well known paper of Benci and 
Cerami [3] where the subcritical case was considered) provided 9 > is small. 
This result was extended for the p-Laplacian operator and p < q < p* by Alves 
and Ding [T] . The results presented here can be viewed as versions of the papers 
[15l [3 [T] for the case of systems. 

As far we know, the first results for homogeneous system like (P) are due 
to Morais Filho and Souto [12] (see also [5]). After this work many results 
have been appeared (see [3 [7J [HI [HI El [HJ HH] and references therein). Among 
then, the most related with our paper if the work of Han [S] , where the author 
considered the case Q(s,t) — ais 2 + a^i 2 and H(s,t) — s a t^ with a + (3 = 2*. 
His results was complemented by Ishiwata in [TQl [11], with different classes 
of homogeneous nonlinearities being considered. Our paper extends and/or 
complements the results found in [TH &\ [5J [TUl E] ■ Although there are some 
multiplicity results for systems like (P) via Ljusternik-Schnirelmann theory, we 
do not know any article that relates the topology of f2 with the number of 
solutions and contains a general class of nonlinearities such as those considered 
here. 

The paper is organized as follows. In Section 2 we present the abstract 
framework of the problem, we prove a local compactness result and obtain the 
existence of one nonnegative solution for (P). Section 3 is devoted to the proof of 
some technical results concerned the properties of sequences which minimize Sh 
and the asymptotic behavior of the minimax levels associated to the problem. 
Theorems 11.11 and 11.21 are proved in Section 4 and we devote the last section for 
some further remarks about examples and possible extensions of the results. 

2 The PS condition and an existence result 

We start this section fixing some notation. We denote Br(0) := {x £ ]& N : 
< R} and by C$°(A) the set of all functions / : A -> K of class C°° with 
compact support contained in the open set A C R N . We denote by \\f\\ p the 
L p -norm of / £ L P (A). In order to simplify the notation, we write J, f instead 
of J A f(x)dx. We also omit the set A whenever A — ft. 

We remark for future reference that, if p > 1 and F is a p-homogeneous 
C 1 -function, then the following holds 

(i) if we set M F := m&x{F(s,t) : s, t £ R, \s\ p + \t\ p = 1} then, for each 
(s, t) £ R 2 , we have that 

\F( S ,t)\<M F (\s\ p + \t\P); (2.1) 
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(ii) \7F is a (p — l)-homogeneous function and, for each (s,t) £ i 2 , we have 
that 

sF s (s,t)+tF t (s,t) = P F(s,t). (2.2) 

We proceed now with the extension of the functions Q and H . Notice that 
(Qi) and (Hi) allow us to give a C 1 extension of Q and H to the whole K 2 as 

Q(s,t):=Q(s+,t + ), H(s 1 t):^H(s + ,t+) 1 (2.3) 

where s + := max{s,0}. In the setting of Theorem 11.21 with Q satisfying (Qi) 
instead of (Qi), the above extension is not differentiable. So, in this case, we 
extend Q in the following way 

Q(s,t) := i+) - VQ(s+ *+) ■ (a",*-), (2.4) 

where s~ = max{— s,0}. For simplicity, we shall write only H to denote the 
extension H . The extension of Q depends on it to satisfy (Qi) or (Qi). In both 
cases, the extension is of class C 1 and will be denoted just by Q. 

By using (|2.1[) and well know arguments, we see that the weak solutions of 
(P) are precisely the critical points of the C 1 -functional ^ : X — > R given by 

ixAz) ■■= \\\4 2 - J QxA*) - ^ J H(z), z e X, 

where X is the Sobolev space Hq(H) x H^fl) endowed with the norm 

\\(u,v)f:= J (|V U | 2 + |V V | 2 ). 

We notice that, in the definition of Ix,^, we are denoting Q\^(z) :— Q{z) for 
z G K 2 . We shall write Qx,^ instead of Q just to emphasize that the smallness 
condition in the statement of the main theorems depends on the value of the 
parameters ^ and A defined in fj 1 . 1 1) - f) 1 . 2j) . 

We introduce the Nehari manifold of by setting 

Mx^ :={zeX\{(0,0)}:I'xAz)z = 0} 

and define the minimax cx 4l as 

CA.p := inf I\Jz). 

In what follows, we present some properties of ca,^ and Mx,^- Its proofs can 
be done as in [171 Chapter 4]. First of all, we note that there exists r = rx.^ > 0, 
such that 

||,z|| > r > for each z G W A , M . (2.5) 

It is standard to check that Ix.^ satisfies Mountain Pass geometry. So, we 
can use the homogeneity of Q and H to prove that ca iAI can be alternatively 
characterized by 

cx,u = inf max /a,u(t(*)) = m f max/ A .u(te) > 0, (2.6) 
7er A , M t S [o,i] ' Mwwy zex\{o} t>o 
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where T x ^ := {7 £ C([0, 1],X) : 7(0) = 0, Ja^CtOO) < 0}. Moreover, for each 
z <E X \ {0}, there exists a unique t 2 > such that t z z G A/a,^. The maximum 
of the function t M> I x>lx (tz), for t > 0, is achieved at t = t z . 

Let £ be a Banach space and J £ C 1 (£',R). We say that (z n ) C £ is a 
Palais-Smale sequence at level c ((PS) C sequence for short) if J(z n ) — > c and 
J'(z n ) -¥ 0. We say that J satisfies (PS) C if any (PS) C sequence possesses a 
convergent subsequence. 

Lemma 2.1 The functional I x ^ satisfies the (PS) C condition for allc < jtS^ 2 



Proof. Let (z n ) = ((u n ,v n )) C X be such that I' x Az n ) — > and I\.^{z r , 
c < jjS^ 2 . The definition of I\ tfi and (|2.1j) provide ci, C2 > such that 



c + a\\z n \\+o n (l) = h^{z n ) - — l' Xfli {z n )z n 



where hereafter o„(l) denotes a quantity approaching zero as n — > 00. The 
above expression implies that (z n ) C X is bounded. So, we may suppose that 
z n — v z := (it, u) weakly in X and z„ — >■ 2: strongly in L q (Q) x L 9 (f2). Moreover, 
a standard argument shows that I' x (z) = 0. 

By setting z„ := (u n ,v n ) — (u n — u, v n — v) we can use the strong convergence 
in L q (Q) x L q (fl) and P21 Lemma 5] to conclude that 



Qa,m(*0 = / Qa, p («) + o„(l), / H(zn) = J H{z) + J H(z n ) + o„(l). 

(2.8) 

This and the weak convergence of (z n ) provide 

c + „(l) = / A , Al (z) + i||z Jl || 2 -l J i||z„f-l J H(z n ), (2.9) 

where we have used I\,^{z) > 0. 

By using I' x „(z n ) — > and (|2.8[) again, we get 



o„(l) = I' Xf2 (z n )z n = \\z n \\ 2 -q J Qx,n(z n ) - J H{z n ) 
= I' x Jz)z + \\z n f- jH(z n ). 
Recalling that I' x (z) = 0, we can use the above equality and (|2.9p to obtain 

lim ||5„|| 2 = 6 = lim [ H(z n ), ±-b = ( \ - ^ ) b < c, 
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for some b > 0. 

In view of the definition of Sh, we have that 

2/2* 

12 > Sh U H{z n ) ' 

Taking the limit we get b > Snb 2 ^ 2 " . So, if b > 0, we conclude that b > S^ 2 
and therefore 

1 s N H ' 2 <lb< c < Is£ /2 , 

TV ff - TV - N H 
which does not make sense. Hence b = and therefore z„ — 2; strongly in JV. 
□ 

Before presenting our next result we recall that, for each e > 0, the function 

p p (N-2)/4 

where Cat := iV(7V - 2)( JV - 2 )/ 4 , satisfies ||V$ £ || 2 = ||$ £ || 2 ; = S"^ 2 , where S* is 
the best constant of the Sobolev embedding X» 1 - 2 (R JV ) i 2 *(R Ar ). Thus, using 
[T2I Lemma 1] and the homogeneity of H, we obtain A, B > such that 

||(^$ £ ,B<i> £ )|| 2 (A 2 +£? 2 ) S N / 2 



/ ff(A$ £ ,£?$ £ ) 



2 / 2 * H{A,B)W ||$ £ ||2, 



from which it follows that 

5ff - 5)2/2* S- (2.11) 

The above equality and the ideas introduced by Brezis and Nirenberg [4] are 
the keystone of the following result. 

Lemma 2.2 Suppose that Q satisfies (Qo), with 2 < q < 2* , and A, fi defined 
in U.l)) - n~Jty) are positive. Then, 



C\.fj, < TtSjj 



N/2 

N~' 



The same result holds if q — 2 and and A, \x £ (0, 9i(Q)/2), where 9\(Q) > 
denotes the first eigenvalue of (— A, Hq (Q)). 

Proof. We consider a nonnegative function 6 Cq^R^) such that 4> = 1 in 
73 fl (0) C fi, ee in R w \ B 2R (0) and define 

<£(a:)$ e (a:) 

w £ (x) — 



110,$ 



£||2* 
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where $ e was defined in (I2.10[) . Since ||to e ||2« = 1, we can use the homogeneity 
of Q and H to get, for any t > 0, 

I KlM {tAw £ ,tBw £ ) = t -(A 2 + B 2 )\\w £ \\ 2 -VQx^BWweWl-t—H&B). 

We shall denote by h e (t) the right-hand side of the above equality and consider 
two distinct cases. 

Case 1. 2 < q < 2*. 

In this case there exists t £ > such that 

h s (t £ ) = max/i e (t). (2.12) 

Let 

9e{t) ■= | (A 2 + B 2 )\\w £ \\ 2 - t-H(A, B), t > 0, 
and notice that the maximum value of g £ occurs at the point 

{A 2 +B 2 )\\w £ \\ 2 \ 1/{2 ~ r) 



t 

So, for each t > 0, 



H(A,B) 



(A< rn 1 ((A 2 + B 2 )\\w e \\ 2 \ N/2 



N \ H(A, B) 2 / 2 * 
and therefore 

MM < ± ( ^^^ ) N ' 2 -tlQxAA,B)\\w e \\l. (2.13) 

We claim that, for some C2 > 0, there holds 

i«Q x>tl (A,B) >c 2 . 

Indeed, if this is not the case, we have that t Sn — > for some sequence e n — > + . 
But it is proved in gj (1.11) and (1.12)] that 

\\w e \\ 2 = S + 0{e {N - 2)/2 ). (2.14) 

Thus, 

< ca, m < sup I\^(tAw en ,tBw En ) = I\^{t £n Aw £n ,t £n Bw £n ) -> 0, 

t>0 

which does not make sense. So, the claim holds and we infer from (|2.13l) and 
(j2T4| that 



he{te) < 



1 / (A 2 + B 2 ^ 



\ \ 1 

t^ + o( £ (Af - 2)/2 )J -C2IWS 



JV \H(A,B) 2 / 2 
< ^ 2 + 0(e( N - 2 y 2 )~c 2 \\ W£ \\l 
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It is proved in QU Claim 2, p. 778] that lim £ ^ 0+ e (2 - N ^ 2 \\w £ \\ q q = +00. Thus, 
we conclude from the above inequality that, for each e > small, there holds 

ca,a< < sup Ix,^{tAw e) BAw e ) = h e (t £ ) < -^- T S% /2 . 

i>0 

Case 2. q = 2. 

In this case we have that h £ {t) = if, and only if, 

(A 2 + B 2 )\\w e \\ 2 -2Q Kfi (A,B)\\w e f 2 = t r - 2 H(A,B). 

Since we are supposing A < #i(fi)/2, we can use Poincare's Inequality to obtain 

2Q Kfl (A,B)\\w s g < 2\{A 2 + B 2 )\\w e \\ 2 

< O^iA 2 + B 2 )\\w £ \\ 2 < (A 2 + B 2 )|KH 2 . 

Thus, there exists t £ > satisfying (|2.12[) . By using the definition of w £ and [4, 
(1.12) and (1.13)] we get 

f £ ( N - 2 )^ + 0(e^ N - 2 ^ 2 ) if iV > 5, 

\\w e g = l (2.15) 
1 e^-^llogel + O^" 2 )/ 2 ) if AT = 4. 

Arguing as in the first case we conclude that, for e > small, there holds 

M*e) < ±S»' 2 + 0(eV<-W) -c 2 \\ We \\ 2 < ^S N H '\ 

where we have used (|2.15[) in the last inequality. This concludes the proof. □ 

Remark 2.3 The previous lemma remains valid if we suppose that N = 3 and 
4 < q < 6. Indeed, it suffices to notice that in this case, according to p. 
779], the function w £ above satisfies lim £ _ > o+ £ ll^ell* = +oo- So, the 

same arguments of Case 1 hold. 

As a byproduct of Lemmas l2.ll and l2.2l we obtain the following generalization 
of [H Theorem 1]. 

Theorem 2.4 Under the hypotheses of Theorem ] 1 . 1\ the problem (P) possesses 
a nonzero nonnegative solution whenever 2 < q < 2* and A, fJ, > 0, or q = 2 
and A, /x £ (0, #i(fi)/2). The same result holds if N = 3 and 4 < q < 6. 

Proof. Since I\ tfl satisfies the geometric conditions of the Mountain Pass The- 
orem, there exists (z n ) C X such that 
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It follows from Lemma 12.11 (with Remark 12.31 in the case TV = 3) and Lemma 
12.21 that (z n ) converges, along a subsequence, to a nonzero critical point z — 
(u,v) el of According to (12. 3[) and (12. 2[) . we have that 

Since z is a critical point and the integral above vanishes, it follows that z~ = 0. 
Hence, u, v > in D, and the theorem is proved. □ 



Theorem 2.5 Under the hypotheses of Theorem \1.2\ the problem (P) possesses 
a nonzero nonnegative solution whenever A, jj, > 0. The same result holds if 
N = 3 and 4 < q < 6. 

Proof. As before, we obtain a nonzero critical point z of A simple calcula- 
tion shows that the extension given in (|2.4I) implies that Q s (s,t) > for s < 0, 
and Qt{s,t) > for t < 0. Hence, using the extension of H and arguing as in 
the previous theorem we obtain 

= I'\A z >~ = -W z ~W 2 - J (Qu{u,v)u- +Q v (u,v)v~) < -\\z-\\ 2 , 
and the result follows. □ 



3 Some technical results 

In this section we denote by Ad(M. N ) the Banach space of finite Radon measures 
over M. N equipped with the norm 

\a\ = sup W(<P)\- 

V GC (R JV ),|| V |U<1 

A sequence (cr n ) C M(M. N ) is said to converge weakly to a e M(M. N ) provided 
cr n (<p) — > o-((p) for all ip G Co(K w ). By the Banach- Alaoglu theorem, every 
bounded sequence (cr n ) C Ai($L N ) contains a weakly convergent subsequence. 

The next result is a version of the Second Concentration-Compactness Lemma 
of P.L. Lions [13l Lemma 1.1]. 

Lemma 3.1 Suppose that the sequence (w n ) C T> 1,2 (WL N ) x T> > 2 (R ) satisfies 
w n w weakly in D 1 ' 2 ^) x 2? 1 > 2 (R A '), 

w„(x) — > w(x) for a.e. x £ M. N , 

V (w n - w)\ 2 (T weakly in M(R N ), 
H(w n — u>) — v f weakly in Ai(M. N ) 
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and define 

n 

R-^oo n-s-oo J| x |>^ ' R-Hx> n ^oo J\ X \>R 



Coo := lim limsup / |Vu>„| 2 dx, := lim limsup / H(w n )dx. 

n^oo J\x\>R 



(3.1) 

Then 

limsup / \\7w n \ 2 dx = \a\ + (Too + / \S7w\ 2 dx, (3.2) 
limsup / H(w n )dx = \v\ + + / H(w)dx, (3-3) 

\v\ 2/2 ' <S^|a| and ^ /2 * < 5ffVoo. (3.4) 

Moreover, if w — and\v\ 2 / 2 = Sg \o~\, then there exists xq, x\ £ suc/i 
£/ia£ v — 5 Xo and a — 5 X1 . 

Proof. We first recall that, in view of the definition of Sh, for each nonnegative 
function (p £ C£°(R W ) we have that 

/ r \ 2//2 * / f \ 2//2 * 

/ y> 2 (x)il(u> n )dx I = / #(</?(x)u; n )dx ) < Sff 1 \\(p(x)w n \\ 2 . 
Moreover, arguing as in [121 Lemma 5], we have that 

tp(x)H(w n — w)dx = / ip(x)H(w n )dx — / ij}(x)H (w)dx + o n (l), 



for each ip £ C^°(M jv ). Since iJ is 2*-homogeneous, we can use the two above 
expressions and argue along the same lines of the proof of [TT1 Lemma 1.40] 
(see also Lemma 2.2]) to conclude that ([33-([3lJ hold. If w = and 
|j/| 2 / 2 = S^lcl the same argument of pTl step 3 of the proof of Lemma 1.40] 
implies that the measures v and a are concentrated at single points xo, x\ £ R N , 
respectively. □ 



Remark 3.2 For future reference we notice that the last conclusion of the above 
result holds even ifw^O. Indeed, in this case we can define w n :— w n —w and 
notice that 

w n -^w = weakly in V 1 ' 2 (R N ) x V 1 ' 2 (R N ), 

w n (x) — > for a.e. x £ K.^, 

V(uj„ - w)\ 2 a weakly in M(R N ), 

H(vj n — w) v weakly in A4(R N ). 

But w n — w = w n — w and therefore a — a and v = v, where a and v are as in 
Lemma \3.1\ Thus, if\v\ 2 ^ 2 = S : H 1 \o\ we also have that\v\ 2 l 2 = S^\a\ and 
the result follows from the last part of Lemma ??. 
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Before stating one of the main results of this section we introduce the fol- 
lowing notation. Given r > 0, y G R N and a function z £ X , we extend z to the 
wholelR^ by setting z{x) := Oiix G R N \£l and define z^' r G H 1 (R N ) x H 1 (R N ) 
as 

z»- r (a;) := r {N - 2 ^ 2 z(rx + y), xeR N . 
Proposition 3.3 Suppose (z n ) C X is such that 

/ H(z n ) = 1 and lim ||z„|| 2 = Sh- 

J n— >oo 

TTien </iere exisi (r n ) c (0, oo) and (y n ) C l w sucft i/iai i/ie sequence (z^™' 1 ™) 
strongly converges to z^O in V 1 ' 2 (R N ) x P 1 ' 2 ^). Moreover, as n — » oo, we 
Aave i/iai r„ — ^ and y n — ¥ y G 0. 

Proof. We first extend z„ by setting z„(a;) := if ie G K N \ SI. For each r > 
we consider 



F n {r) := sup / H(z n ). 

V m N JB r (y) 

Since lim r ^o-Fn( r ) = and linv^oo F n (r) = 1, there exist r„ > and a se- 
quence (2/^)fceN C I 1 *' satisfying 



- = F n (r n ) = lim / iT(zn) 

2 fc ^°°JB_fyi) 



Recalling that lim^i^oo /g ^H(z n ) = we conclude that (y„) is bounded. 
Hence, up to a subsequence, lim^oo y\ = y n G and we obtain 



Br„(S») 



We shall prove that the sequences (r„) and (y n ) above satisfy the statements 
of the lemma. First notice that 

\ = I H(z n ) = / H( Z y^) = sup / HW«). (3.5) 

If we denote w n := z^™ ,r ™, a straightforward calculation provides 

lim ||w„|| 2 = lim ||z„|| 2 = S H , / i?(w„) = 1. 

Hence, we can apply Lemma I3TT1 to obtain u; G i? 1 (R'' v ) x if 1 (M 7V ) satisfying 

Sir=M +^oo + \\wf, l=\v\+v OB + [ #(«>), (3.6) 
I 1 2 / 2 * < S]; 1 ] <j\ and < S^a^. (3.7) 
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The second equality above implies that / H(w), |^| , 6 [0, 1]. If one of 
these values belongs to the open interval (0,1), we can use (|3.6I) . 2/2* < 1, 
(jH(w)) 2 / 2 ' < S^IMI 2 and (g^J) to get 



Sh = Sn[\v\+u 0o + / H{w) 

< S H (\v\ 2/r + vU 2 ' + (J H{w) \ ' \<S H , 



which does not make sense. Thus J H(w), \is\, v x s {0, 1}. Actually, it follows 
from (|3.5p that §\ x \ >R H(w n ) < 1/2 for any R > 1. Thus, we conclude that 
Voc = 0. 

Let us prove that \v\ = 0. Suppose, by contradiction, that \v\ = 1. It 
follows from the first equality in (|3.7p that Sh < | o | . On the other hand, the 
first equality in Q3.6P provides \ a\ < Sh- Hence, we conclude that | cr | = Sh. 
Since we are supposing that |^| = 1 we obtain \ v\ 2 ^ 2 = S , ^ 1 |cr|. It follows from 
Remark [321 that v = 6 Xo for some xq £ M. N . Thus, from (|3.5p . we get 



~ > lim / H{w n ) = I du = \p\ = 1. 



2 



n— >oo 



This contradiction proves that \ u\ = 0. 

Since |^| = foe = we have that J RN H(w) — 1. This and (|3.6p provide 



lim |K|| 2 = S H >|M| 2 >Sif( / 



2/2* 

# M ) = Sh. 



So, ||w|| 2 = S H and therefore w n -> to ^ strongly in D 1 ' 2 (R JV ) x P 1 ' 2 (R iV ) 
and w„(a;) — > w;(a;) for a.e. x e R w . 

In order to conclude the proof we notice that 

I|w„||l2(R«) xL 2( R ]V) = — \\Zn\\L 2 (n)xL 2 (n)- 

Since (z n ) is bounded and w ^ 0, we infer from the above equality that, up to 
a subsequence, r n — > ro > 0. If \y n \ — > oo we have that, for each fixed x € K , 
there exists fijfN such that r„.T + y n f2 for n > n x . For such values of n we 
have that w n {x) = 0. Taking the limit and recalling that x € K is arbitrary, we 
conclude that w = 0, which is absurd. So, along a subsequence, j/„->j/e l w . 

We claim that r = 0. Indeed, suppose by contradiction that r > 0. Then, 
as n becomes large, the set Q n := (fi — y n )/r n approaches to fi := {Q — y)/r Q ^ 
R . This implies that u> has compact support in M. N . On the other hand, since 
w achieves the infimum in (I1.3P and H is homogeneous, we can use the Lagrange 
Multiplier Theorem to conclude that w = (u, v) satisfies 



— Am = XH u (u, v), — Av — XH v (u, v), x € 
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for A = 2Sh/2* > 0. It follows from (H3) and the Maximum Principle that at 
least one the functions u, v is positive in M . But this contradicts suppw C fio- 
Hence, we conclude that tq = 0. Finally, if y $ £1 we obtain r n x + y n $ fi for 
large values of n, and therefore we should have w = again. Thus, y £ f2 and 
the proof is finished. □ 

We finalize this section with the study of the asymptotic behavior of the 
minimax level ca, m as both the parameters approaches zero. 

Lemma 3.4 We have that 

V 1 qN/2 

hm c x ,„ = co,o = -t-S h . 

Proof. We first prove the second equality. It follows from A = fi = that 
Qo.o = 0. If A, B, w e , g e and t £ are as in the proof of Lemma 12.11 we have that 
(t e Aw s ,t e Bw e ) e7V ,o- Thus 

1 f (A 2 + B 2 ) ^ N/2 



co,o < h,v{teAw E , t e Bw e ) = — ^ H ^ s)2/2 , 



JV \ H(A, B) 2 / 2 * 

Taking the limit as e — > + and using (|2.1ip . we conclude that co,o < jqS^ 2 . 

In order to obtain the reverse inequality we consider (z n ) C X such that 
Io,o(z n ) — > co,o and Iq (z n ) — > 0. The sequence (z„) is bounded and therefore 
^o,o( z n) z " = Iknll 2 _ J = o„(l). It follows that 



lim ||z„|| 2 = 6= lim / H(z n ). 

n— foo n — >oo / 



2 /2* 

Taking the limit in the inequality Sh (J H{z n )j < II II 2 we conclude, as in 
the proof of Lemma I2TT1 that Ncq.q = b > S^ 2 . Hence, 

c 0:0 = hm h,o{z n ) = lim ( ]-\\z n \\ 2 - — [ H(z n )\ = —b > ^S^ /2 , 

and therefore co,o = jjS^ 2 . 

We proceed now with the calculation of lim A M _ >0 + c^^. Let (A„), (/i„) Cl + 
be such that A n , /i„ — > + . Since /i„ defined in (jl.ip is positive, we have that 
Qa„./j„(- z ) > whenever z is nonnegative. Thus, for this kind of function, we 
have that I\ n ,^ n {z) < Iq,o(z). It follows that 

c a„,m„ = mf max/ A „. M „(te) 
z#(0,0) *>o 

< inf max /a u (i^) 

z^(0,0),2>0 t>0 

< inf max Jo a(tz) = Cq 0, 

z#(0,0),z>0 t>0 
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where we have used, in the last equality, that the infimum co,o is attained at a 
nonnegative solution. The above inequality implies that 

limsupcA„, Mn < c ,o- (3.8) 

n^QG 

On the other hand, it follows from Theorem 12.41 that there exists (z n ) = 
(it n , v n ) C X such that 

Since ca„.^„ is bounded, the same argument performed in (|2.7[) implies that 
(z n ) is bounded in X. Thus J Qx n ,^ n {z n ) < X n J(\u n \ q + \v n \ q ), from which it 
follows that 



lim / Q\ ntl * n (z n )=0. (3.9) 
Let f„ > be such that t n z n G A/o.o- Since z„ G A/"a„./j„, we have that 



Co,0 < hAtnZn) = h n ,» n (t n Zn) +t%, J Qx n ,/J, n ( z ri) 

Zn ) 

= CA„, M „ +t nJ Qx n ,n n {Zn)- 

If (i„) is bounded, we can use the above estimate and (|3.9[) to get 



c o,o < liminf c\ n 



This and (J3T8J) proves the lemma. 

It remains to check that (t n ) is bounded. A straightforward calculation 
shows that 

Since z n G A/"a„,^„ we obtain 

Iknll 2 = Q I Qx^M + I H(z n ) < o„(l) + S H 2 ' /2 \\z n \f . 



Hence ||z n || 2 > Ci > 0, and therefore it follows from the above expression that 
J H(z n ) > C2 > 0. This, the boundedness of (z n ) and (13. 10)) imply that (t n ) is 
bounded. The lemma is proved. □ 
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4 Proof of the main theorems 



From now on we fix r > such that the sets 

0+ := {ieR": dist(>, ft) < r}, Q~ := {x € CL : dist(x, dfl) > r} 
are nomotopic equivalents to f2. We define the functional 

■= \\\ Z W 2 ~ J Q*A Z ) - 2^ j H ( Z )' z ^ X r,rad, 

where X rrac i :— {(u,v) : u, v G Hq(B 7 .(0)) and u, w are radial functions}. 
We denote by Mx,^ its associated Nehari manifold and set 

m\ t u := inf J\,u(z). 

According to )12\ Lemma 1] the infimum Sh can be attained by functions be- 
longing to D 'JM. N ) x T) r ' ad (R N ). So, arguing as in the proof of Lemma I3T41 and 
Theorems 12.41 and 12.51 . we obtain the following result. 



Lemma 4.1 Under the hypothesis of Theorem \l.l[ the infimummx,^ is attained 
by a positive radial function zx.^ 6 X r>ra ^ whenever 2 < q < 2* and A, fi > 0. 
or q — 2 and A, /i£ (0, 4p). Moreover 

m x u < -77 Sh^ 2 an d nm m A u = -77 S^ 2 . 



TTie same result hold if we assume the hypothesis of Theorem \1.2\ and A, /j, > 0. 
Moreover, both results hold if N = 3 and 4 < q < 6. 

We introduce the barycenter map fix, ^ : A/"a,/j — > K N as follows 

:= -^72 / H(z)xdx. 
S H J 

This maps has the following property. 

Lemma 4.2 There exists A* > such that Px,^{z) £ whenever z e A/a,^, 
A, /i £ (0, A*) and Za.mO 2 ) < rnx,^. 

Proof. Suppose, by contradiction, that there exist (A n ), (fi n ) C R + and (w n ) C 
J^x n ,tm such that A„, fi n -)■ 0+ as n 00, Ix n ,p, n (w n ) < toa„, m „ but Px m ,» n ( w n) $ 

r/2 

Standard calculations show that (uvO is bounded in X. Moreover 



= I\ n ,ii n ( w n)iv n = \\w n \\ 2 - q I Qx„,p, n (w n ) - / H(w n ) 
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As in the proof of Lemma [3.41 we have that J Qx n ,iM n (w n ) — > and therefore 
lim„^oo \\w n \\ 2 = linin-s.oc J H(w n ) = b > 0. Notice that 

Recalling that ca„, m „ and toa„,^„ both converge to jjS^ 2 , we can use the above 
expression and J Qa„,^„ (dh) — > again to conclude that b = S^ 2 , that is, 

lim || w „|| 2 = ^ /2 = lim [ H(w n ). (4.1) 

n— >-oo 71— ^oo J 

Let t„ := (J H{w n ))^ 1 ^ 2 > and notice that z n := £ n ti; n satisfies the 
hypotheses of Proposition 13.31 Thus, for some sequences (r„) C (0, oo) and 
(y n ) C R N satisfying r„->0,y„->yel]we have that z£"< r ™ -> z in 2? 1 ' 2 (R JV ) x 
D 1 ' 2 (R JV ). 

The definition of z n , f|4. 1 [) . the strong convergence of (z^"' r ™) and the Lebesgue's 
Theorem provide 

P\ n ,iJ.n( w n) = -%/2 I H(z n )xdx = (l + o n (l)) jH{z n )xdx 

S Tjr J J 

= (1 + o n (l)) / H{zl^){r n x + y n ) dx 



= (l + o n (l)) (J H(z)ydx + o n (l) 

Since y £ Q and / H(z) = 1, the above expression implies that 

lim dist(/3\ ntl i n (w n ),Sl) = 0, 

n— >oo 

which contradicts P\ n ^ n (w n ) £ The lemma is proved. □ 

According to Lemma [4.1[ for each A, /i > small the infimun mj )(1 is attained 
by a nonnegative radial function Z\ tll . We consider 

and define the function ~f\^ : Q~ —> I™*'" by setting, for each y £il~, 

j z\,n{x -y) if a € B r (y), 
otherwise. 



A change of variables and straightforward calculations show that the map 7a, m 
is well defined. Since z\^ is radial, we have that J B ^ H(z\^)x dx = 0. Hence, 
for each y £ Q~, we obtain 
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where 



If we define F Al/1 : [0, 1] x (Af Xjfl n 7™*'") -> by 
we have the following. 

Lemma 4.3 There exists A** > smc/i that, 

F Xtlt ([o,i\x (Wx ifl n/^)) eft 

whenever A, /i 6 (0, A**). 

Proof. Arguing by contradiction, we suppose that there exist sequences (A„), (/x n ) C 
R+ and (t n ,z n ) G [0,1] x (A/"a„, m „ n C^'"") such that A «> M« -> 0+ 



r > 



as 

n — s- oo, and F Xnjfln (t n , z n ) £ f2+. Up to a subsequence t„ — >• to G [0,1]. 
Moreover, the compactness of fl and Lemma 14.21 imply that, up to a subse- 
quence, P\ n ^ n {z n ) ->j/£ f2^y 2 C n+. We claim that a(A„,^„) > 1 . If this is 
true, we can use the definition of F to conclude that F\ n u n (t n , z n) V £ ft!~j 
which does not make sense. 

It remains to check the above claim. It follows from Lemma r4.1l that 

z JB,.(0) z JB r (0) ly 

As before J B , sQ\ n ,n n (z\ n ,n n ) ~> °- Triis ; ^A n ,/i„( 2 A n ,^„) = 0, the above ex- 
pression and the same arguments used in the proof of Lemma 14.11 imply that 



Iim I H{z Xnflln ) = S% / \ 

n— >oo J 



The equality above and the definition of a(A, /i) imply that a(A„, jji n ) — > 1. The 
lemma is proved. □ 



Corollary 4.4 Let A := min{A*,A**} > 0, with A* and A** given by Lemmas 
\4-%\ and \4-S\ respectively. If Q is such that A,/i G (0, A) i/ien 

cat /m ,, M (/™^") > catn(fi). 

Proof. It suffices to use Lemmas 14.21 and 14.31 and argue as in [TJ Lemma 4.3]. 
We omit the details. □ 

We are now ready to prove our main results. 
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Proof of Theorems \l.l\ and \1.2\ Let A > be given by Corollary |4.4l and suppose 
that Q is such that A, \i e (0,A). Using Lemma [2TT1 and arguing as in [1, 
Lemma 4.2] we can prove that the functional I\ tll restricted to N\.^ satisfies the 
{PS) C condition for all c < jjS^ 2 . Since mj i(1 < jfS^ 2 , standard Ljusternik- 
Schnirelmann theory provides cat (I A critical points of the constrained 

functional. If z 6 A/x ijU is one of these critical points, the same argument of [TJ 
Lemma 4.1] shows that z is also a critical point of the unconstrained functional, 
and therefore a nontrivial solution of (P) . As before, the obtained solutions are 
nonnegative in ft. The results follow from Corollary 14.41 □ 



5 Some further remarks 

We start this last section presenting some functions which satisfy our hypothe- 
ses. We have the following example from [12j . Let 2 < q < 2* and 

k 

P q {s,t) := ais q + a 2 t q +^hs a 't s} \ s,t>0, 

i=l 

where Oii,fii > 1, oti+fii = q and ax, «2, bi £ R. The following functions and its 
possible combinations, with appropriated choices of the coefficients ax, a,2, bi, 
satisfy our hypotheses on Q 

Q(s,t) = P q (s,t), Q(s,t) = y 'P rq (s,t) and Q(s,t) = ^ffi 

with I > 0. Hence, we see that our subcritical term is more general than those 

of [anniE]. 

The form of H is more restricted due to (#4). This technical condition has 
already appeared in [T^] and it is important to guarantee that the constant 
Sh defined in p. 31) does not depend on O. As quoted in [T2], the concavity 
condition (H4) is satisfied if H e C 2 (R^,R) is such that H st (s,t) > for each 
(s,t) S R 2 + . 

Although we have more restrictions on the shape of H, it can have the 
polynomial form 

H(s,t)=P 2 *(s,t). 

Thus, differently from [SJ [TUJ [IT], we can deal here with functions H which 
possesses coupled and no coupled terms. For example, the function 

H(s, t) = ais T + a 2 t 2 ' + a 3 s Q ^, 

with ai € R, a, (3 > 1, a + j3 = 2* satisfies the hypotheses (Ho) — (H4) for 
appropriated choices of the coefficients a^. We also mention that the positivity 
condition in (H2) can holds even if some of the coefficients ai are negative. As a 
simple example, suppose that H is as above with ax, a,i > and 03 < 0. Since 
s a v P < s 2 _|_ t 2 , the condition (H2) holds for 03 > max{— ax, —02}- 
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Another interesting remark is that we can obtain versions of our theorems 
by interchanging conditions like (Qi) and (Qi) for both the functions Q and H. 
More specifically, let us consider the following assumption 

(Hi) H S (0,1) > and H t (l,0) > 0. 

A simple inspection of our proofs shows that Theorem 1 1.1 1 is valid if we suppose 
(Hi) and (Qi). The same is true for Theorem 11.21 This last theorem is also 
true if we suppose (Hi) and (Qi). The difference among these various settings 
relies in the form of the possible coupled terms. 

A simple inspection of our proofs show that, instead of just one subcritical 
term, we can consider in (P) a subcritical nonlinear term of the form 

k 
i=l 

with each function Qi being ^-homogeneous, 2 < qi < 2*, and satisfying the 
same kind of hypotheses of Q. In this case, for each i = 1, . . . , k, we define the 
numbers /i;, Xi as in (jl.ip ~ p.2p . and the results hold if max^i^. ^!/^, A^} is 
small enough. 

With some additional conditions we can assure that the solutions obtained 
in this paper are positive. Indeed, if we suppose that 

(Q2) Q s (s,t) > 0, Q t (s,t) > for each (s,t) 6 M.\, 

we can apply the Maximum Principle in each equation of (P). Thus, if (u, v) is 
a nonnegative solution, then u = or u > in f2, the same holding for v. We 
need only to discard solutions of the type (u, 0) or (0,v). This can be done if 
we guarantee some kind of strongly coupling for the system. In what follows, 
we present some situations where this can be done. 

If we are under the conditions of Theorem ll.il we assume a stronger form of 
(Qi) and (Hi), namely that VQ(1, 0) = VQ(0, 1) = VH (1, 0) = VH(0, 1) = 0. 
In this way, if (it, 0) is a solution then 

= I'xJu,0)(u,0) = -\\u\\ 2 ~ J (q u (u,0)u+^H u (u,0)u 

and therefore u = 0. Analogously, if (0,v) is a solution then v = 0. In the 
setting of Theorem 11.21 and considering the solution (u, 0) we obtain, from the 
second equation, that 

= Q v (u, 0) + H v (u, 0) = u 9 ~ l Q v (l, 0). 

Since from (Qi) we have that Q v (l, 0) > 0, it follows that u = 0. The argument 
for (0, v) is analogous. 
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